Workflow [PR], commit [a7bf1bd]

@codex review

@blau-ai

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 567457bd95

CI triage

3 failing checks — none is a code failure introduced by this PR. Breakdown: 1 policy (DCO), 1 pre-existing/branch-wide (Iceberg regression), 1 likely flaky (Swarms regression). Everything else (Fast test, Builds, Quick functional, Source upload, Config) is green.

1. DCO — ⚠️ PR-related, but a sign-off policy issue (not a code bug)

None of the 10 commits on this branch carry a Signed-off-by: trailer, so the DCO app fails the check.

Fix: the commits need to be re-signed, e.g. rebase the branch with sign-off:

git rebase --signoff antalya-26.3
git push --force-with-lease

(and configure git config user.name/user.email to match the committer). I can't do this myself — it requires a force-push to rewrite history, which is outside my permissions. This is orthogonal to the test results below.

2. Regression release iceberg_2 — ✅ NOT caused by this PR (pre-existing)

All failures are under /iceberg/export partition/** — the export partition transaction & lossy-cast tests:

✘ /iceberg/export partition/{no,ice,glue} catalog/transactions/commit survives pre-publish failure (non-retry cleanup)
✘ /iceberg/export partition/{no,ice,glue} catalog/casting/lossy/UInt64 narrows to Int32 without allow_lossy_cast
✘ /iceberg/export partition/{no,ice,glue} catalog/casting/out_of_bounds/...

Failing assertion is assert last_status == expected_status in the export-partition test harness.

Evidence it's not this PR:

• This PR touches only distributed-query-planning code (150 files under src/Processors/QueryPlan/**, planner, exchanges). It does not touch Iceberg, object-storage writes, or the cast logic these tests exercise.
• The identical set of export partition scenarios fails on the unrelated PR #2086 (an OAuth fix, merged) and other recent PRs. This is a branch-wide breakage in the actively-developed export partition feature (cf. Plain merge tree partition export #2032, Fix Iceberg REST catalog commit safety under post-commit network failure #2003, Change export partition log levels #2004), not a regression from Backport of #106020: Multi-stage distributed queries #1999.

Next step: ignore for the purposes of gating this PR; it should be tracked/fixed on antalya-26.3 independently.

3. Regression release swarms — 🟡 Most likely flaky

Only 1 of 1520 scenarios failed:

✘ /swarms/feature/node failure/check restart swarm node   (1m 52s)
    AssertionError: assert r.exitcode == exitcode, error(r.output)

This is a node-restart resiliency scenario — it kills/restarts a swarm node and asserts on a shell command's exit code, which is inherently timing-sensitive.

Evidence it's flaky rather than PR-caused:

• The Swarms suite is green on PRs Antalya 26.3: Fixes for OAuth --login flow #2086, Plain merge tree partition export #2032, and Datalake catalog auth token profile events #2010 (recent runs, both before and after this run) — it is not broken branch-wide.
• Only this single restart scenario tripped; the other 1519 passed.
• The multi-stage distributed feature this PR adds is experimental and disabled by default, so it should not be on the default swarm read path.

Next step: re-run the Swarms job. If it fails again on the same scenario, the most plausible interaction point is the "Shrink idle thread pools of the distributed query infrastructure" change (backport of ClickHouse#106978), which alters distributed-infra thread-pool lifecycle — worth checking against swarm-node restart timing. I couldn't fetch the full raw.log/report.html from S3 (no network access in my environment), so the exact failing command is inferred from the job log rather than read directly.

Bottom line: no evidence this PR breaks any test. Get the commits signed off (DCO), re-run Swarms to clear the flake, and don't block on iceberg_2 — that's a pre-existing antalya-26.3 issue unrelated to this backport.

@ianton-ru could you pls comment on swarm fail? is it really unrelated?

The pretty-formatted report is gone, but you can see the failure in the GH job itself: https://github.com/Altinity/ClickHouse/actions/runs/29262641137/job/86882806205?pr=1999

Codex Review: Didn't find any major issues. Nice work!

Thank you!

@ianton-ru could you pls comment on swarm fail? is it really unrelated?

The pretty-formatted report is gone, but you can see the failure in the GH job itself: https://github.com/Altinity/ClickHouse/actions/runs/29262641137/job/86882806205?pr=1999

@zvonand I can't because test artifacts were removed.
Try to merge to retrigger tests

@zvonand
Found (your link is openable).

    Assertion values
      assert r.exitcode == exitcode, error(r.output)
             ^ is <testflows.connect.shell.Command object at 0x706199dbd730>
      assert r.exitcode == exitcode, error(r.output)
             ^ is = 0
      assert r.exitcode == exitcode, error(r.output)
                           ^ is 138
      assert r.exitcode == exitcode, error(r.output)
                        ^ is = False
      assert r.exitcode == exitcode, error(r.output)
      ^ is False

Yes, it's a known thing, unrelated to PR.

Codex found nothing, and expecting a careful manual review of a backport that has 11000 LoC would be a little ambitious :)
the CI is green, codex is happy -- I think this one can be verified and merged? @ianton-ru wdyt?

Codex found nothing, and expecting a careful manual review of a backport that has 11000 LoC would be a little ambitious :) the CI is green, codex is happy -- I think this one can be verified and merged? @ianton-ru wdyt?

Yeah, backports were without large conflicts, I think can be merged.